Looks like this will match any record of compatibility exception or approved exception anywhere in docs/workflows, so if there's a line in the README that says, "There is no approved exception on file," it'll pass.

Should we gate this on something more explicit? Suggestions:

Option A: An HTML comment in the docs

<!-- vip:compatibility-exception approved -->
An HTML comment is handy here because markdown doesn't render it — it's invisible in the displayed README but sits in the source, so it reads as a deliberate machine marker, not prose. The check becomes anchored and specific:
const EXCEPTION_MARKER = /<!--\s*vip:compatibility-exception\s+approved\s*-->/i;
if (EXCEPTION_MARKER.test(ctx.docsText)) {
  return { ...base, status: 'pass', message: '…' };
}

Option B: A structured field in composer.json (preferred)

{
  "extra": {
    "vip": { "compatibility-exception": "approved" }
  }
}

const flag = ctx.composer?.extra?.vip?.['compatibility-exception'];

if (flag === 'approved') { … pass … }

Also, instead of auto-passing for this rule, should we require it to still route to the human-review section for sign-off or maybe downgrade to a warning that says "exception claimed, needs reviewer confirmation?"

This checks out ref: trunk up top, so it always builds and publishes trunk's code — but the release is created with --target "$GITHUB_SHA". On a workflow_dispatch, GITHUB_SHA is the commit of whatever ref you triggered from, which the UI lets you pick. So if someone dispatches from a feature branch, we publish trunk to npm but tag vX.Y.Z on the feature branch commit. The tag ends up pointing at code that was never shipped.

Should we tag the tree we actually built? --target "$(git rev-parse HEAD)" after checkout would keep the tag and the published code in sync.

The config guard (rule 5) passes if is_array( / is_ready( / is_available( appears anywhere in the concatenated PHP source, and rule 9's telemetry guard does the same with class_exists(. Those are generic functions used all over a normal plugin, so an unrelated is_array() in some helper makes it print "Config access is guarded against a missing or invalid constant" even when the actual config read has no guard at all. It's checking "does this word exist in the codebase," not "is the risky call guarded."

Could we scope the search to a window around the config constant (and around record_event( for rule 9)? Same approach the secret-key scan just below already uses.

Rule 6 tells "valid" from "incomplete" by counting keys. If one doc example has fewer keys than another, it calls that pair valid+incomplete. But fewer keys usually just means fewer optional settings, not a missing required field, so two perfectly valid examples of different lengths pass as if one were the broken case the rule is trying to require.

Detecting "missing a required field" statically is genuinely hard, so maybe we lean on the explicit signal instead. The mentionsIncomplete regex is already there. Require the docs to actually label the incomplete example rather than guessing from key count?

If the clone dies partway (network drop) or scaffoldTree throws, we leave a half-populated target on disk. The next init then hits the "Target directory is not empty" guard and the user is stuck cleaning up by hand and the error doesn't say why. The comment at line 86 says we fail fast so we don't leave a half-laid-down directory behind, but that only covers the name validation, not these two spots.

Can we wrap the clone + scaffold in a try/catch and rmSync(target, { recursive: true, force: true }) on failure, so a failed run leaves nothing behind?

Same here: https://github.com/Automattic/integration/pull/1/changes#r3619936703

The skip set only excludes .png/.jpg/.jpeg/.gif/.webp, but we read every other file as utf8 and write it back. If the kit ships any other binary (.ico, .woff2, .ttf, .pdf) that happens to contain a token's ASCII bytes, reading it as utf8 swaps the invalid bytes for U+FFFD and we write the corrupted version back. The updated !== contents check saves token-free binaries, so it only bites a binary that also contains a token — but the comment says we skip "binary assets" and we only skip some of them.

Could we either broaden the skip list to the other binary types, or sniff for a NUL byte and skip anything that looks binary before rewriting?

The PHP versions are checked with .includes('8.4'), so it matches that string anywhere in the workflow YAML, not just on a PHP-version line. CI files are full of other version numbers — node-version: 18.4, mysql:8.4, a pinned action tag — any of those makes it think PHP 8.4 is covered. So an integration that only tests 8.2 but has a Node matrix of 18.3/18.4/18.5 reports full 8.2–8.5 coverage and passes.

The WP checks right above already dodge this with \b6.9\b. A word boundary here helps (\b8.4\b won't match inside 18.4), but it still matches mysql:8.4, so ideally we scope the match to a php-version key the way the wpLatest pattern scopes to a WP key. Can we tie it to the php line instead of a loose substring?